Calibration of depth sensing using a sparse array of pulsed beams

ABSTRACT

Depth sensing apparatus includes a radiation source, which is configured to emit a first plurality of beams of light pulses toward a target scene. An array of a second plurality of sensing elements is configured to output signals indicative of respective times of incidence of photons on the sensing element, wherein the second plurality exceeds the first plurality. Light collection optics are configured to image the target scene onto the array of sensing elements. Processing and control circuitry is coupled to receive the signals from the array and is configured to search over the sensing elements in order to identify, responsively to the signals, respective regions of the array on which the light pulses reflected from the target scene are incident, and to process the signals from the identified regions in order determine respective times of arrival of the light pulses.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/532,517, filed Aug. 6, 2019, which claims the benefit of U.S.Provisional Patent Application 62/803,612, filed Feb. 11, 2019, and U.S.Provisional Patent Application 62/809,647, filed Feb. 24, 2019. Thisapplication is related to U.S. patent Application Ser. No. 16/532,513,filed Aug. 6, 2019, entitled “Depth sensing using a sparse array ofpulsed beams.” All of these related applications are incorporated hereinby reference.

FIELD OF THE INVENTION

The present invention relates generally to systems and methods for depthmapping, and particularly to beam sources and sensor arrays used intime-of-flight sensing.

BACKGROUND

Existing and emerging consumer applications have created an increasingneed for real-time three-dimensional (3D) imagers. These imagingdevices, also known as depth sensors, depth mappers, or light detectionand ranging (LiDAR) sensors, enable the remote measurement of distance(and often intensity) to each point in a target scene—referred to astarget scene depth—by illuminating the target scene with an optical beamand analyzing the reflected optical signal. A commonly-used technique todetermine the distance to each point on the target scene involvestransmitting one or more pulsed optical beams towards the target scene,followed by the measurement of the round-trip time, i.e. time-of-flight(ToF), taken by the optical beams as they travel from the source to thetarget scene and back to a detector array adjacent to the source.

Some ToF systems use single-photon avalanche diodes (SPADs), also knownas Geiger-mode avalanche photodiodes (GAPDs), in measuring photonarrival time. For example, U.S. Pat. No. 9,997,551, whose disclosure isincorporated herein by reference, describes a sensing device thatincludes an array of SPAD sensing elements. Each sensing elementincludes a photodiode, including a p-n junction, and a local biasingcircuit, which is coupled to reverse-bias the p-n junction at a biasvoltage greater than the breakdown voltage of the p-n junction by amargin sufficient so that a single photon incident on the p-n junctiontriggers an avalanche pulse output from the sensing element. A biascontrol circuit is coupled to set the bias voltage in different ones ofthe sensing elements to different, respective values.

U.S. Patent Application Publication 2017/0176579, whose disclosure isincorporated herein by reference, describes the use of this sort ofvariable biasing capability in selectively actuating individual sensingelements or groups of sensing elements in a SPAD array. For thispurpose, an electro-optical device includes a laser light source, whichemits at least one beam of light pulses, a beam steering device, whichtransmits and scans the at least one beam across a target scene, and anarray of sensing elements. Each sensing element outputs a signalindicative of a time of incidence of a single photon on the sensingelement. (Each sensing element in such an array is also referred to as a“pixel.”) Light collection optics image the target scene scanned by thetransmitted beam onto the array. Circuitry is coupled to actuate thesensing elements only in a selected region of the array and to sweep theselected region over the array in synchronization with scanning of theat least one beam.

SUMMARY

Embodiments of the present invention that are described hereinbelowprovide improved depth mapping systems and methods for operating suchsystems.

There is therefore provided, in accordance with an embodiment of theinvention, depth sensing apparatus, including a radiation source, whichis configured to emit a first plurality of beams of light pulses towarda target scene. An array of a second plurality of sensing elements isconfigured to output signals indicative of respective times of incidenceof photons on the sensing element, wherein the second plurality exceedsthe first plurality. Light collection optics are configured to image thetarget scene onto the array of sensing elements. Processing and controlcircuitry is coupled to receive the signals from the array and isconfigured to search over the sensing elements in order to identify,responsively to the signals, respective regions of the array on whichthe light pulses reflected from the target scene are incident, and toprocess the signals from the identified regions in order determinerespective times of arrival of the light pulses.

In some embodiments, the radiation source includes at least onevertical-cavity surface-emitting laser (VCSEL), and may include an arrayof VCSELs. Additionally or alternatively, the sensing elements includesingle-photon avalanche diodes (SPADs).

In some embodiments, the processing and control circuitry is configuredto group the sensing elements in each of the identified regions togetherto define super-pixels, and to process together the signals from thesensing elements in each of the super-pixels in order to determine therespective times of arrival. In a disclosed embodiment, the processingand control circuitry includes multiple processing units, wherein eachof the processing units is coupled to process the signals from arespective one of the super-pixels. Additionally or alternatively, theprocessing and control circuitry is configured, after identifying therespective regions, to actuate only the sensing elements in each of theidentified regions, while the remaining sensing elements in the arrayare inactive.

In some embodiments, the processing and control circuitry is configuredto identify the respective regions of the array on which the lightpulses reflected from the target scene are incident by selecting a setof candidate regions in which the reflected light pulses are likely tobe incident, and performing an iterative search over the array startingwith each of the candidate regions while operating the radiation sourceand receiving the signals to find the regions of the array onto whichthe light pulses reflected from the target scene are incident. In oneembodiment, the processing and control circuitry is configured toidentify the candidate regions based on nominal design values togetherwith assembly tolerances and operational tolerances of the apparatus.The processing and control circuitry may be configured to perform theiterative search by shifting repeatedly from those sensing elements fromwhich no timing signal is received to neighboring sensing elements,until a number of the regions from which the signals are receivedexceeds a preset threshold.

In other embodiments, the processing and control circuitry is configuredto identify the respective regions of the array onto which the lightpulses reflected from the target scene are incident by finding aninitial set of candidate regions of the array on which the light pulsesreflected from the target scene are incident, calculating a model, basedon the initial set, that predicts locations of additional regions of thearray on which the light pulses reflected from the target scene areexpected to be incident, and searching over the locations predicted bythe model while operating the radiation source and receiving the signalsin order to identify additional regions of the array on which the lightpulses reflected from the target scene are incident. In a disclosedembodiment, the processing and control circuitry is configured to adjustthe model iteratively until a number of the regions from which thesignals are received exceeds a preset threshold. The processing andcontrol circuitry may be configured to adjust the model by adding theidentified additional regions to the initial set to produce a new set ofthe regions, and updating the model based on the new set.

Alternatively or additionally, the model is selected from a group oftypes of models consisting of a homographic model, a quadratic model,and a low-order spline.

There is also provided, in accordance with an embodiment of theinvention, a method for depth sensing, which includes driving aradiation source to emit a first plurality of beams of light pulsestoward a target scene. The target scene is imaged onto an array of asecond plurality of sensing elements, configured to output signalsindicative of respective times of incidence of photons on the sensingelement, wherein the second plurality exceeds the first plurality. Asearch is performed over the sensing elements in order to identify,responsively to the signals, respective regions of the array on whichthe light pulses reflected from the target scene are incident. Thesignals from the identified regions are processed in order determinerespective times of arrival of the light pulses.

The present invention will be more fully understood from the followingdetailed description of the embodiments thereof, taken together with thedrawings in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic side view of a depth mapping system, in accordancewith an embodiment of the invention;

FIG. 2A is a schematic side view of a radiation source used in the depthmapping system of FIG. 1, in accordance with an embodiment of theinvention;

FIG. 2B is a schematic frontal view of an array of emitters used in theradiation source of FIG. 2A, in accordance with an embodiment of theinvention;

FIG. 2C is a schematic frontal view of an array of emitters that can beused in the radiation source of FIG. 2A, in accordance with anotherembodiment of the invention;

FIG. 3A is a schematic representation of a pattern of spots projectedonto a target scene, in accordance with an embodiment of the invention;

FIG. 3B is a schematic frontal view of a ToF sensing array, inaccordance with an embodiment of the invention;

FIG. 3C is a schematic detail view of a part of the ToF sensing array ofFIG. 3B, onto which images of the spots in a region of the target sceneof FIG. 3A are cast, in accordance with an embodiment of the invention;

FIGS. 4A and 4B are schematic frontal views of a ToF sensing arrayshowing sets of super-pixels that are selected for activation andreadout in two different time periods, in accordance with an embodimentof the invention;

FIG. 5 is a block diagram that schematically illustrates circuitry forprocessing of signals from a super-pixel, in accordance with anembodiment of the invention;

FIG. 6 is a flowchart that schematically illustrates a method foridentifying the pixels on a SPAD array that receive laser spots, inaccordance with an embodiment of the invention; and

FIG. 7 is a flowchart that schematically illustrates a method foridentifying the pixels on a SPAD array that receive laser spots, inaccordance with another embodiment of the invention.

DETAILED DESCRIPTION OF EMBODIMENTS Overview

In some of the embodiments described in the above-mentioned U.S. PatentApplication Publication 2017/0176579, SPADs are grouped together into“super-pixels,” wherein the term “super-pixel” refers to a group ofmutually-adjacent pixels along with data processing elements that arecoupled directly to these pixels. At any instant during operation of thesystem, only the sensing elements in the area or areas of the array thatare to receive reflected illumination from a beam are actuated, forexample by appropriate biasing of the SPADs in selected super-pixels,while the remaining sensing elements are inactive. The sensing elementsare thus actuated only when their signals provide useful information.This approach reduces the background signal, thus enhancing thesignal-to-background ratio, and lowers both the electrical power needsof the detector array and the number of data processing units that mustbe attached to the SPAD array.

One issue to be resolved in a depth mapping system of this sort is thesizes and locations of the super-pixels to be used. For accurate depthmapping, with high signal/background ratio, it is important that thesuper-pixels contain the detector elements onto which most of the energyof the reflected beams is imaged, while the sensing elements that do notreceive reflected beams remain inactive. Even when a static array ofemitters is used, however, the locations of the reflected beams on thedetector array can change, for example due to thermal and mechanicalchanges over time, as well as optical effects, such as parallax.

In response to this problem, some embodiments of the present inventionprovide methods for calibrating the locations of the laser spots on theSPAD array. For this purpose, processing and control circuitry receivestiming signals from the array and searches over the sensing elements inorder to identify the respective regions of the array on which the lightpulses reflected from the target scene are incident. Detailed knowledgeof the depth mapping system may be used in order to pre-compute likelyregions of the reflected laser spots to be imaged onto the SPAD array. Arandom search in these regions will converge rapidly to the correctlocations of the laser spots on the array. Alternatively oradditionally, a small subset of the locations of laser spots can beidentified in an initialization stage. These locations can be used insubsequent iterative stages to predict and verify the positions offurther laser spots until a sufficient number of laser spots have beenlocated.

Even following meticulous calibration, it can occur in operation of thedepth mapping system that some of the pixels or super-pixels on whichlaser spots are expected to be imaged fail to output usable timingsignals. In some cases, ancillary image data can be used to identifyareas of the scene that are problematic in terms of depth mapping, andto recalibrate the super-pixel locations when necessary. This ancillaryimage data can be provided, for example, by a color image sensor, whichcaptures two-dimensional (2D) images in registration with the SPADarray.

The emitter arrays used in the embodiments described below are “sparse,”in the sense that the number of pulsed beams of optical radiation thatare emitted toward a target scene is substantially less than the numberof pixels (i.e., SPADs or other sensing elements) in the array thatreceives the radiation reflected from the scene. The illumination poweravailable from the emitter array is projected onto a correspondinglysparse grid of spots in the scene. The processing and control circuitryin the apparatus then receives and processes signals only from thepixels onto which these spots are imaged in order to measure depthcoordinates.

The pixels onto which the spots are imaged are referred to herein as the“active pixels,” and the “super-pixels” are made up of groups ofadjacent active pixels, for example 2×2 groups. The pixels in the arraythat fall between the active pixels are ignored, and need not beactuated or read out at all, as they do not contribute to the depthmeasurement and only increase the background level and noise.Alternatively, a different number, such as one, two, three or morepixels, may be included in a super-pixel. Furthermore, although theembodiments described herein relate specifically to rectangularsuper-pixels, the group of SPAD pixels in a super-pixel may have adifferent shape, such as, for example, diamond shape, triangular,circular, or irregular. The exact location of the spot within the SPADpixels varies slightly depending on the distance to the scene due to asmall amount of parallax. At any given time, the signals from the SPADpixels of the super-pixel are processed together in measuring for agiven laser spot both its strength (intensity) and its time of flight.Additionally, the signals from the SPAD pixels may be processed asindividual signals for determining the location of the laser spot withinthe super-pixel.

An advantage of using a sparse emitter array is that the available powerbudget can be concentrated in the small number of projected spots,rather than being spread over the entire field of view of the sensingarray. As a result of this concentration of optical power in a smallnumber of spots, the signal levels from the corresponding activepixels—and thus the accuracy of ToF measurement by these pixels—areenhanced. This signal enhancement is particularly beneficial forlong-range depth measurements and for depth mapping in conditions ofstrong ambient light, such as outdoors.

The concentration of optical power in a sparse array of spots can befurther enhanced by arranging the emitters in multiple banks, andactuating these banks in alternation. The laser beams generated by theemitters are typically collimated by a collimating lens and may bereplicated by a diffractive optical element (DOE) in order to increasethe number of projected spots. The pulses of optical radiation emittedfrom the different banks of the emitters are incident, after reflectionfrom the target scene, on different, respective sets of the activepixels. The processing and control circuitry can then receive andprocess the signals from the active pixels in these respective sets insynchronization with actuation of the corresponding banks of emitters.Thus, during any given period in the operation of the apparatus, theprocessing and control circuitry need receive and process the signalsonly from one active set of sensing elements, while all other setsremain inactive. This sort of multi-bank, synchronized operation makesit possible to time-multiplex processing resources among the differentsets of sensing elements, and thus reduce circuit complexity and powerconsumption.

Because the spots reflected from the target scene are imaged sparselyonto the SPAD array, the number of possible super-pixels is much largerthan the number of laser spots, and only a small fraction of the totalnumber of pixels in the SPAD array should be active at any given timeand coupled to a processing unit for the purpose of measuringtime-of-flight. Therefore, information is required as to which of theSPAD super-pixels to activate at any given time.

A mapping of SPAD pixels to processing units, i.e., the assignment ofSPAD pixels to super-pixels, may be determined initially during afactory calibration. However, temperature changes during operation, aswell as mechanical shocks, may alter the mechanical parameters of themapping, thus modifying the positions of the laser spots on the SPADarray and necessitating recalibration during operation in the field. Anexhaustive search could be used to determine which of the SPAD pixels toconnect to the processing units, wherein all pixels are searched todetect laser spots; but this approach suffers from at least two basicproblems:

-   -   Some laser spots may not fall on objects in the scene or may        fall on objects that absorb the laser wavelength, thus returning        no pulse. Therefore a search may not always be successful.    -   As the distribution of laser spots is very sparse when compared        to the number of pixels of the SPAD array, exhaustive search        will require a large number of exposures and will take a long        time.

The embodiments of the present invention that are described hereinaddress these problems by providing improved methods for calibrating thelocations of the laser spots on the SPAD array. These methods can beapplied not only in the sorts of arrays that are shown in the figuresand described hereinbelow, but also in other SPAD-based systems, such assystems comprising multiple banks of SPADs, as well as SPADs of varioussizes, and systems using various sorts of emitters and emitter arrays,including emitters whose beams are replicated by a DOE. The presentmethods can then be extended, mutatis mutandis, to multi-bank systems,by activating the SPAD pixels and performing the calibration bank bybank.

In a disclosed embodiment, detailed knowledge of the depth mappingsystem is utilized to pre-compute likely regions of the reflected laserspots to be imaged onto the SPAD array. A search in these regions, forexample a random search, will converge rapidly to the correct locationsof the laser spots on the array.

Another disclosed embodiment uses a two-stage solution: in aninitialization stage, a small subset of the locations of laser spots isidentified, and in a subsequent iterative stage, the positions offurther laser spots are predicted by a model and verified. Iterativesteps of spot detection are utilized to refine the model and addlocations, until a sufficient number of laser spots have been located.

System Description

FIG. 1 is a schematic side view of a depth mapping system 20, inaccordance with an embodiment of the invention. Depth mapping system 20comprises a radiation source 21, which emits M individual beams (forexample, M may be on the order of 500). The radiation source comprisesmultiple banks of emitters arranged in a two-dimensional array 22 (asshown in detail in FIG. 2B), together with beam optics 37. The emitterstypically comprises solid-state devices, such as vertical-cavitysurface-emission lasers (VCSELs) or other sorts of lasers orlight-emitting diodes (LEDs). The beam optics typically comprise acollimating lens and may comprise a diffractive optical element (DOE,not shown), which replicates the actual beams emitted by array 22 tocreate the M beams that are projected onto the scene 32. (For example,an array of four banks of pixels with 16 VCSELs in a 4×4 arrangement ineach bank may be used to create 8×8 beams, and a DOE may split each beaminto 3×3 replicas to give a total of 24×24 beams.) For the sake ofsimplicity, these internal elements of beam optics 37 are not shown.

A receiver 23 in system 20 comprises a two-dimensional SPAD array 24,together with J processing units 28 and select lines 31 for coupling theprocessing units to the SPADs, along with a combining unit 35 and acontroller 26. SPAD array 24 comprises a number of detector elements Nthat is much larger than M, for example, 100×100 pixels or 200×200pixels. The number J of processing units 28 depends on the number ofpixels of SPAD array 24 to which each processing unit is coupled, aswill be further described with reference to FIG. 4.

Array 22, together with beam optics 37, emits M pulsed beams of light 30towards a target scene 32. Although beams 30 are depicted in FIG. 1 asparallel beams of constant width, each beam diverges as dictated bydiffraction. Furthermore, beams 30 diverge from each other so as tocover a required area of scene 32. Scene 32 reflects or otherwisescatters those beams 30 that impinge on the scene. The reflected andscattered beams are collected by objective optics 34, represented by alens in FIG. 1, which form an image of scene 32 on array 24. Thus, forexample, a small region 36 on scene 32, on which a beam 30 a hasimpinged, is imaged onto a small area 38 on SPAD array 24.

A Cartesian coordinate system 33 defines the orientation of depthmapping system 20 and scene 32. The x-axis and the y-axis are orientedin the plane of SPAD array 24. The z-axis is perpendicular to the arrayand points to scene 32 that is imaged onto SPAD array 32.

For clarity, processing units 28 are shown as if separate from SPADarray 24, but they are commonly integrated with the SPAD array.Similarly, combining unit 35 is commonly integrated with SPAD array 24.Processing units 28, together with combining unit 35, comprise hardwareamplification and logic circuits, which sense and record pulses outputby the SPADs in respective super-pixels, and thus measure the times ofarrival of the photons that gave rise to the pulses, as well as thestrengths of the optical pulses impinging on SPAD array 24.

As further described below in reference to FIG. 4, processing units 28together with combining unit 35 may assemble histograms of the times ofarrival of multiple pulses emitted by array 22, and thus output signalsthat are indicative of the distance to respective points in scene 32, aswell as of signal strength. Circuitry that can be used for this purposeis described, for example, in the above-mentioned U.S. PatentApplication Publication 2017/0176579. Alternatively or additionally,some or all of the components of processing units 28 and combining unit35 may be separate from SPAD array 24 and may, for example, beintegrated with controller 26. For the sake of generality, controller26, processing units 28 and combining unit 35 are collectively referredto herein as “processing and control circuitry.”

Controller 26 is coupled to both radiation source 21 and receiver 23.Controller 26 actuates the banks of emitters in array 22 in alternationto emit the pulsed beams. The controller also provides to the processingand combining units in receiver 23 an external control signal 29, andreceives output signals from the processing and combining units. Theoutput signals may comprise histogram data, and may be used bycontroller 26 to derive both times of incidence and signal strengths, aswell as a precise location of each laser spot that is imaged onto SPADarray 24.

To make optimal use of the available sensing and processing resources,controller 26 identifies the respective areas of SPAD array 24 on whichthe pulses of optical radiation reflected from corresponding regions oftarget scene 32 are imaged by lens 34, and chooses the super-pixels tocorrespond to these areas. The signals output by sensing elementsoutside these areas are not used, and these sensing elements may thus bedeactivated, for example by reducing or turning off the bias voltage tothese sensing elements. Methods for choosing the super-pixels initiallyand for verifying and updating the selection of super-pixels aredescribed, for example, in the above-mentioned provisional patentapplications.

External control signal 29 controls select lines 31 so that eachprocessing unit 28 is coupled to a respective super-pixel, comprisingfour SPAD pixels, for example. The control signal selects thesuper-pixels from which the output signals are to be received insynchronization with the actuation of the corresponding banks ofemitters. Thus, at any given time, processing units 28 and combiningunit 35 read and process the signals only from the sensing elements inthe areas of SPAD array 24 that receive the reflected pulses from scene32, while the remaining sensing elements in the array are inactive. Theprocessing of the signals from SPAD array 24 is further described inreference to FIG. 4. For the sake of simplicity, the detailed structuresof emitter array 22 and SPAD array 24 are not shown in FIG. 1.

For clarity, the dimensions of emitter array 22 and SPAD array 24 havebeen exaggerated in FIG. 1 relative to scene 32. The lateral separationof emitter array 22 and SPAD array 24, referred to as the “baseline,” isin reality much smaller than the distance from emitter array 22 to scene32. Consequently a chief ray 40 (a ray passing through the center ofobjective optics 34) from scene 32 to SPAD array 24 is nearly parallelto rays 30, leading to only a small amount of parallax.

Of those M super-pixels that are activated and coupled to the Jprocessing units 28, either all of them or a subset of m super-pixels,wherein m <M, will receive a reflected laser beam 30. The magnitude of mdepends on two factors:

-   -   1. The calibration of SPAD array 24, i.e., the choice of the M        super-pixels, and    -   2. The number of laser beams 30 that are actually reflected from        scene 32.        The value M may correspond to the total number of emitters when        all of the emitters are actuated together, or to the number of        emitters in each bank when the banks are actuated in        alternation, as in the present embodiments.

Even if all M laser beams 30 were to be reflected from scene 32, m willbe less than M if SPAD array 24 is not properly calibrated. (Calibrationprocedures described in the above-mentioned provisional patentapplications can be used to maximize m.) Consequently, controller 26will receive signals indicating times of arrival and signal strengthsfrom only m processing units 28. Controller 26 calculates from thetiming of the emission of beams 30 by VCSEL array 22 and from the timesof arrival measured by the m processing units 28 the time-of-flight ofthe m beams, and thus maps the distance to the corresponding m points onscene 32.

Controller 26 typically comprises a programmable processor, which isprogrammed in software and/or firmware to carry out the functions thatare described herein. Alternatively or additionally, controller 26comprises hard-wired and/or programmable hardware logic circuits, whichcarry out at least some of the functions of the controller. Althoughcontroller 26 is shown in the figure, for the sake of simplicity, as asingle, monolithic functional block, in practice the controller maycomprise a single chip or a set of two or more chips, with suitableinterfaces for receiving and outputting the signals that are illustratedin the figure and are described in the text.

One of the functional units of controller 26 is a depth processing unit(DPU) 27, which processes signals from both processing units 28 andcombining unit 35, as will be further described below. DPU 27 calculatesthe times of flight of the photons in each of beams 30, and thus mapsthe distance to the corresponding points in target scene 32. Thismapping is based on the timing of the emission of beams 30 by emitterarray 22 and from the times of arrival (i.e., times of incidence ofreflected photons) measured by processing units 28. Controller 26typically stores the depth coordinates in a memory, and may output thecorresponding depth map for display and/or further processing.

Emitter Array

FIG. 2A is a schematic side view of radiation source 21, in accordancewith an embodiment of the invention. VCSEL array 22 comprises anintegrated circuit chip on which multiple banks of VCSELs are formed (asshown in FIG. 2B, for example). The VCSELs emit respective beams 30toward optics 37, which collimate and project the beams toward thetarget scene. Optics 37 optionally comprise a diffractive opticalelement (DOE), which splits the optical radiation emitted by each of theVCSELs into multiple beams 30, for example a 3×3 array of beams.

To enable selection and switching among the different banks, array 22 ismounted on a driver chip 50, for example, a silicon chip with CMOScircuits for selecting and driving the individual VCSELs or banks ofVCSELs. The banks of VCSELS in this case may be physically separated,for ease of fabrication and control, or they may be interleaved on theVCSEL chip, with suitable connections to driver chip 50 to enableactuating the banks in alternation. Thus, beams 30 likewise irradiatethe target scene in a time-multiplexed pattern, with different sets ofthe beams impinging on the respective regions of the scene at differenttimes.

FIG. 2B is a schematic frontal view of array 22 used in beam source 21,in accordance with an embodiment of the invention. Array 22 in thisexample comprises eight banks 52, with seventy-two emitters 54, such asVCSELs, in each bank. In this case, array 22 generates 578 beams.

FIG. 2C is a schematic frontal view of an array 60 of vertical emitters54 that can be used in beam source 21 in place of array 22, inaccordance with another embodiment of the invention. In this case fourbanks 62 a, 62 b, 62 c and 62 d of emitters 54 are interleaved asalternating vertical stripes on a substrate 64, such as a semiconductorchip: Each bank comprises a number of stripes that alternate on thesubstrate with the stripes in the other banks. Alternatively, otherinterleaving schemes may be used.

As further alternatives to the pictured embodiments, array 22 maycomprise a larger or smaller number of banks and emitters. Typically,for sufficient coverage of the target scenes with static (non-scanned)beams, array 22 comprises at least four banks 52 or 62, with at leastfour emitters 54 in each bank, and possibly with a DOE for splitting theradiation emitted by each of the emitters. For denser coverage, array 22comprises at least eight banks 52 or 62, with twenty emitters 54 or morein each bank. These options enhance the flexibility of system 20 interms of time-multiplexing of the optical and electrical power budgets,as well as processing resources.

Super-Pixel Selection and Actuation

FIG. 3A is a schematic representation of a pattern of spots 70 ofoptical radiation that are projected onto target scene 32, in accordancewith an embodiment of the invention. Each spot 70 is cast by acorresponding beam 30 (FIG. 1). In the present embodiments, differentgroups of spots 70 are projected onto scene 32 in alternation,corresponding to the alternating actuation of the corresponding banks 52of emitters 54 (FIG. 2B).

FIG. 3B is a schematic frontal view of SPAD array 24 onto which targetscene 32 is imaged, in accordance with an embodiment of the invention.The sensing elements, such as SPADs, in array 24 are too small to beseen in this figure. Rather, FIG. 3B shows the locations of spots 72that are reflected from target scene 70 and imaged onto array 24 by lens34. In other words, each spot 72 is the image on array 24 of acorresponding spot 70 that is projected onto scene 32 by emitter array22. Lens 34 images a region 74 of target scene 32 (FIG. 3A), includingspots 70 that the area contains, onto a corresponding area 76 on array24.

FIG. 3C is a schematic detail view of area 76 of array 24, showing thelocations of spots 72 that are imaged onto the array, in accordance withan embodiment of the invention. These spots 72 may be imaged at the sametime, if they originate from the same bank of emitters, or at different,alternating times if they are from different banks. In this view shownin FIG. 3C, it can be seen that array 24 comprises a matrix of sensingelements 78, such as SPADs. (As noted earlier, sensing elements 78 in anarray are also referred to as “pixels.”) Controller 26 assigns eachprocessing unit 28 to a super-pixel 80 comprising a 2'2 group of thesensing elements 78. In this example, it is assumed that during aninitial calibration stage, spots 72 were imaged onto array 24 atlocations 72 a. Controller 26 thus selected the sensing elements 78 toassign to each super-pixel 80 so as to maximize the overlap between thecorresponding spot 72 and the super-pixel, and thus maximize the signalreceived from each super-pixel.

At some later stage, however, spots 72 shifted to new locations 72 b onarray 24. This shift may have occurred, for example, due to mechanicalshock or thermal effects in imaging device 22, or due to other causes.Spots 72 at locations 72 b no longer overlap with super-pixels 80 inarea 76, or overlap only minimally with the super-pixels. Sensingelements 78 on which the spots are now imaged, however, are inactive andare not connected to any of processing units 28. To rectify thissituation, controller 26 may recalibrate the locations of super-pixels80, as described in the above-mentioned provisional patent applications.

FIGS. 4A and 4B are schematic frontal views of SPAD array 24 showingsets of super-pixels 80 a and 80 b that are selected for activation andreadout in two different time periods, in accordance with an embodimentof the invention. In the embodiment, the selection of the set ofsuper-pixels is synchronized with the selection of banks of emitters.Specifically, assuming array 60 (FIG. 2C) is used for generating thespots on the target scene, super-pixels 80 a will be used when bank 62 ais actuated, and super-pixels 80 b will be used when bank 62 b isactuated (and so forth for banks 62 c and 62 d). Thus, during any givenperiod in the operation of the system 20, processing units 28 serve onlythe one active set of super-pixels 80, while all other sets remaininactive, in an integrated time-multiplexing scheme.

FIG. 5 is a block diagram that schematically illustrates the processingof signals from a super-pixel 80, in accordance with an embodiment ofthe invention. In the pictured embodiment, super-pixel 80 comprises foursensing elements 78 of SPAD array 24. Each processing unit 28 comprisesone or more time-to-digital converters (TDC) 143, wherein the TDCs arehardware elements translating the avalanche events (signals from a SPADpixel due to a detected photon) from each sensing element 78 totime-of-arrival information. Each processing unit 28 further comprises,for each TDC 143, a weight 144, and may comprise a histogram unit (notshown), wherein the time-of-arrival information is aggregated intohistograms, generally over thousands of pulses from VCSEL array 22. Inthe present embodiment, however, the histograms are aggregated centrallyfor super-pixel, without individual histogram units in processing units28.

In FIG. 5, each processing unit 28 is coupled to a single sensingelement 78, thus requiring one TDC 143. Alternatively, processing unit28 may be coupled to two or more sensing elements 78, and will thencomprise a number of TDC's 143 equal to the number of pixels. Forexample, if each processing unit 28 is coupled to four SPAD pixels 24,the number of TDCs 143 per processing unit will be four, and J=4*M.Additionally or alternatively, as noted earlier, processing units 28 maybe switched among different sensing elements 78, which are activated atdifferent, alternating times, in synchronization with the alternatingactuation of the corresponding banks 52 of emitters 54 (FIG. 2B).

The time-of-arrival information from the four processing units 28 isaggregated by combining unit 35, using weights 144, to yield a singlehistogram 146 for super-pixel 80. This combined histogram 146 is sent toDPU 27, which in turn detects, based on histogram 146, whether anyobject or structure was detected in scene 32 by super-pixel 80 and, ifso, reports its depth information based on time-of-flight data.

Additionally, the respective numbers of events reported by the fourprocessing units 28 may be separately summed in combining unit 35 over apredefined interval of arrival times to yield an indication of thereceived signal strength for that interval for each sensing element 78.Typically the interval is configured to start after the end of aso-called “stray pulse” and continue to the end of the histogram. Astray pulse is a pulse that is generated within system 20 as a resultof, for example, an imperfect coating of an optical surface, whichcauses a reflection of the pulses emitted by VCSEL array 22 directlyback into the optical path to SPAD array 24. It is typically anundesired pulse, but one that is very difficult to eliminate altogether.The stray pulse may be utilized for calibrating the timing signals asfollows: A time of arrival of a stray pulse is recorded and subtractedfrom a subsequent timing signal due to a laser pulse that has beenreflected by scene 32. This subtraction yields a relative time-of-flightfor the received laser pulse, and compensates for any random firingdelays of VCSEL array 22, as well as for most of the VCSEL and SPADdrifts related to temperature changes.

These four indicators of signal strength are also transferred to DPU 27(in conjunction with combined histogram 146). The indicators may be usedby DPU 27 to determine a precise location of the spot on sensingelements 78.

In some embodiments, the four units of TDC 143, as well as combiningunit 35, reside in the same chip as SPAD array 24, while the rest ofsignal processing, including DPU 27, resides in separate controller 26.A major reason for generating single combined histogram 146 forsuper-pixel 80 is to reduce the information that is transferred fromSPAD array 24 to DPU 27 and to controller 26. The partitioning into twoseparate units reflects the fact that SPAD array 24 and the associatedunits perform primarily optical and analog functions, while controller26 performs mostly digital and software-driven operations.

Super-Pixel Calibration by Search in Precomputed Regions

FIG. 6 is a flowchart that schematically illustrates a method foridentifying the pixels in a sensing array that receive laser spots, inaccordance with an embodiment of the invention. This method isdescribed, for the sake of convenience and clarity, with reference toSPAD array 24 and the other elements of system 20 (FIG. 1). The methodcan be performed, for example, each time system 20 is turned on.Alternatively, the method can be carried out prior to an initial use ofsystem 20, and the results can be stored for future use. The method canthen be repeated periodically and/or when system performance indicatesthat recalibration may be required.

Alternatively, however, the principles of this method may be applied,mutatis mutandis, in other depth mapping systems of similarconfiguration. For example, VCSEL array 22 could be replaced by a singlelaser (or a small number of lasers), with a beamsplitting element, suchas a diffractive optical element (DOE), to split the laser output intomultiple beams. As another example, other types of sensing arrays,comprising other sorts of detector elements, could be used in place ofSPADs. The method of FIG. 7, as described below, is similarly applicablenot only to system 20, but to other depth mapping systems, as well.

In the method of FIG. 6, three input steps: a nominal design value step150, an assembly tolerance step 152, and an operational tolerance step154, provide inputs for a pre-computation step 156. Design value step150 provides the nominal system design values for depth mapping system(FIG. 1); assembly tolerance step 152 provides the assembly tolerancesof the depth mapping system; and operational tolerance step 154 providesexpected operational tolerances, such as variations of ambienttemperature and the effects of a mechanical shock on the depth mappingsystem.

The above inputs include multiple parameters. For example, a typicalfocal length of collection lens 34 has a nominal value of 2 mm, anassembly tolerance of 0.1 mm and an operational tolerance of 0.05 mm.Each tolerance is normally distributed around zero, with a standarddeviation equal to the above tolerance. The probability distribution ofthe focal length is a normal distribution combined from the two normaldistributions and centered at the nominal value of 2 mm. An additionalexample of a parameter is the baseline between VCSEL array 22 and SPADarray 24. The multiple parameters, such as the two examples describedabove, allow controller 26 to model accurately the optical path taken bythe laser pulses and thus calculate the locations where the spotsimpinge on SPAD array 24.

Based on these inputs, controller 26 calculates a search region for eachof the M laser spots expected on SPAD array 24 (FIG. 1), in apre-computation step 156. Each search region includes a group of pixelsfor which a probability of receiving a laser beam reflected from scene32 is estimated to be higher than a preset threshold, such as 99.9%. Asan example of the calculations performed by controller 26, an increaseof 1% in the focal length of collection lens 34 magnifies the image onSPAD array 24 by 1%, thus shifting the spots in an outward radialdirection. The probability distribution of this parameter and all otherparameters of the input translates to a region around each of thenominal spot locations on SPAD array 24 in which there is a probabilityhigher than 99.9% to find the spot.

Once the search regions have been chosen in pre-computation step 156,controller 26, in a random iterative search step 158, fires a successionof pulses of beams 32 from VCSEL array 22 (FIG. 1), and at the same timeperforms random searches within the search regions to identify the Msuper-pixels that receive the pulsed beams. Alternatively, controller 26may apply other search strategies, not necessarily random, within thesearch regions. During step 158, each processing unit 28 is coupled toreceive signals from a different pixel following each laser pulse orsequence of multiple pulses, and controller 26 checks, using DPU 27,which pixels have output signals due to an incident photon, and whichhave not. Based on the results, controller 26 selects the pixels toinclude in each super-pixel as those on which the photons were found tobe incident. In simulations, the search was found to converge within asuccession of 8-10 repeated sequences of pulsed beams 32 and thusidentify the M super-pixels of SPAD array 24 that receive the M beams.

Once controller 26 has found the M super-pixels, it finishes the searchand assigns, in an assignment step 160, these super-pixels for use in 3Dmapping of scene 32 by depth mapping system 20.

Two-Stage Solution for Super-Pixel Calibration

FIG. 7 is a flowchart that schematically illustrates a two-stage methodfor identifying the super-pixels in SPAD array 24 (FIG. 1) that receivelaser spots, in accordance with another embodiment of the invention. Thefirst stage starts by providing, in an input step 200, a small number m₀of potential process candidates, wherein, in this context, the term“process candidate” is used for those SPAD super-pixels likely toreceive laser spots. A typical number for potential process candidatesis either a fixed number, such as m₀=5, or a percentage, such as 10%, ofthe number M. These potential process candidates may be obtained, forexample, from a previous use of depth mapping system 20.

These potential candidates are coupled to respective processing units28, in a candidate processing step 202. In a first detection step 204,controller 26 fires a sequence of pulses of beams 32 from VCSEL array 22and queries processing units 28 and combining unit 35 to find out howmany of the mo process candidates on SPAD array 24 reported “hits,”i.e., output signals indicating that they had received photons. In afirst comparison step 206, controller 26 checks whether the number ofreported hits in first detection step 204 exceeds a first presetthreshold, for example 8% of M (if initially 10% of M were selected asprocess candidates).

If the number of hits was below the threshold, controller 26 searches,in a search step 208, for hits in the areas around the processcandidates by firing successive pulsed beams 32 from VCSEL array 22 andperforming a single pixel search around the candidates. After new hitshave been identified, the previous process candidates in processcandidate step 202 are replaced by the new hits, and steps 204 and 206are repeated until the number of detected hits in first comparison step206 exceeds the first preset threshold.

The detected hits in first comparison step 206 are used by controller 26to build a model in a modeling step 210. The model expresses thedeviation of the locations of the hits in SPAD array 24 relative totheir nominal locations, i.e., the locations where the reflected laserbeams were expected to be incident on SPAD array 24 according to thedesign geometry of system 20, for example. The model may be a quadraticmodel, a simplified pinhole camera model, or a homographic model, forexample, and it may take into account system tolerances as previouslydescribed.

A homographic model h is an eight-parameter transformation (h₁, . . . ,h₈), mapping a point p=(x, y) to another point p′=(x′, y′) through therelation:

$\left( {x^{\prime},y^{\prime}} \right) = \left( {\frac{{h_{1}x} + {h_{2}y} + h_{3}}{{h_{7}x} + {h_{8}y} + 1},\frac{{h_{4}x} + {h_{5}y} + h_{6}}{{h_{7}x} + {h_{8}y} + 1}} \right)$

The coordinates x and y refer to Cartesian coordinate system 33 ofFIG. 1. Such a model represents the correct spot locations on SPAD array24 in a case where scene 34 comprises a plane (for instance, a wall).

A quadratic model is given by:

x′=a ₁ +b ₁ x+c ₁ y+d ₁ x ² +e ₁ y ² +f ₁ xy

y′=a ₂ +b ₂ x+c ₂ y+d ₂ x ² +e ₂ y ² +f ₂ xy

The equations of a simplified pinhole camera model are as follows: Givena point (x,y,z) in Cartesian coordinate system 33, we first compute theundistorted image coordinates:

${x_{u} = {{f\frac{X}{Z}} + c_{x}}},{y_{u} = {{f\frac{Y}{Z}} + {c_{y}.}}}$

We then apply a distortion operation to obtain the final imagecoordinates:

x _(d) =c _(x)+(x_(u) −c _(x))·p(r), y _(d) =c _(y)+(y _(u) −c_(y))·p(r),

where

r=√{square root over ((x _(u) −c _(x))²+(y _(u) −c _(y))²)}

and p(r)=1+k₁r²+k₂r⁴+k₃r⁶ is a distortion polynomial. The parameters ofthe model are, therefore, the following constants: f, c_(x), c_(y), k₁,k₂, k₃ (see G. Bradski and A. Kaehler, Learning OpenCV, 1^(st) edition,O'Reilly Media, Inc., Sebastopol, Calif., 2008).

Additionally or alternatively, other models, such as splines or moreelaborate models that describe the optics of system 20 to a higherdegree of complexity, may be employed.

Based on one of the above models, controller 26 predicts the locationsof a number of new process candidates, by applying the model to thenominal locations of a number of additional pixels where other reflectedlaser beams were expected to be incident, in a candidate addition step212, making now up a total of m₁ process candidates. Typically, m₁increases in each iteration at candidate addition step 212. In a seconddetection step 214 controller 26 fires an additional sequence of pulsesof beams 30 from VCSEL array 22 and queries how many of the m₁ processcandidates on SPAD array 24 have reported hits.

In a second comparison step 216, controller 26 compares the relativenumber of hits (the ratio between the hits and the total number M ofpulsed beams 30) to a second preset threshold. This latter threshold istypically set to a high value, corresponding to a situation in which thelarge majority of beams 30 are successfully received by correspondingsuper-pixels. If the relative number of hits is less than the secondpreset threshold, controller 26 adjusts the model in modeling step 210based on the detected hits. The adjustment of the model includesrecalculating the model coefficients, as well as, where required, anincrease in the complexity of the model. In an iterative process, of 5-8loops, for example, controller 26 adds new process candidates based onthe model in candidate addition step 212, queries hits in seconddetection step 214, and compares their relative number to the secondpreset threshold in second comparison step 216. As long as the relativenumber of hits does not exceed the second preset threshold, controller26 keeps looping back to model step 210, improving the model based onthe new hits.

Once the relative number of detected hits exceeds the second presetthreshold, controller 26 finishes the search and assigns, in anassignment step 218, the detected hits for use in 3D mapping of scene 32by depth mapping system 20.

In case the number of process candidates at step 214 does not increaseat a given stage of the iteration and is still too low, controller 26may initiate a single-pixel offset search in an offset search step 222.In offset search step 222, a search for the yet undetected laser spotsis performed with a single-pixel offset around their expected locations.

It will be appreciated that the embodiments described above are cited byway of example, and that the present invention is not limited to whathas been particularly shown and described hereinabove. Rather, the scopeof the present invention includes both combinations and subcombinationsof the various features described hereinabove, as well as variations andmodifications thereof which would occur to persons skilled in the artupon reading the foregoing description and which are not disclosed inthe prior art.

1. Depth sensing apparatus, comprising: a radiation source, which isconfigured to emit a first number of beams of light pulses toward atarget scene; an array of a second number of sensing elements,configured to output signals indicative of respective times of incidenceof photons on the sensing element, wherein the first number is greaterthan one, and the second number exceeds the first number; lightcollection optics configured to image the target scene onto the array ofsensing elements; and processing and control circuitry, which is coupledto receive the signals from the array and is configured to select a setof the sensing elements in respective candidate regions of the array inwhich the reflected light pulses are likely to be incident, and toperform an iterative search over the sensing elements in the arraystarting with each of the candidate regions while operating theradiation source and receiving the signals, in order to identify,responsively to the signals, respective regions of the array on whichthe beams of light pulses reflected from the target scene are incident,and to process the signals from the sensing elements in the identifiedregions in order determine respective times of arrival of the lightpulses.
 2. The apparatus according to claim 1, wherein the radiationsource comprises at least one vertical-cavity surface-emitting laser(VCSEL).
 3. The apparatus according to claim 2, wherein the at least oneVCSEL comprises an array of VCSELs.
 4. The apparatus according to claim1, wherein the sensing elements comprise single-photon avalanche diodes(SPADs).
 5. The apparatus according to claim 1, wherein the processingand control circuitry is configured to group the sensing elements ineach of the identified regions together to define super-pixels, and toprocess together the signals from the sensing elements in each of thesuper-pixels in order to determine the respective times of arrival. 6.The apparatus according to claim 5, wherein the processing and controlcircuitry comprises multiple processing units, wherein each of theprocessing units is coupled to process the signals from a respective oneof the super-pixels.
 7. The apparatus according to claim 1, wherein theprocessing and control circuitry is configured, after identifying therespective regions, to actuate only the sensing elements in each of theidentified regions, while the remaining sensing elements in the arrayare inactive.
 8. The apparatus according to claim 1, wherein theprocessing and control circuitry is configured to identify the candidateregions based on nominal design values together with assembly tolerancesand operational tolerances of the apparatus.
 9. The apparatus accordingto claim 8, wherein the processing and control circuitry is configuredto perform the iterative search by shifting repeatedly from thosesensing elements from which no timing signal is received to neighboringsensing elements, until a number of the regions from which the signalsare received exceeds a preset threshold.
 10. A method for depth sensing,comprising: driving a radiation source to emit a first number of beamsof light pulses toward a target scene; imaging the target scene onto anarray of a second number of sensing elements, configured to outputsignals indicative of respective times of incidence of photons on thesensing element, wherein the first number is greater than one, and thesecond number exceeds the first number; selecting a set of the sensingelements in respective candidate regions of the array in which the beamsof light pulses reflected from the target scene are likely to beincident; searching iteratively over the sensing elements in the arraystarting with each of the candidate regions while operating theradiation source and receiving the signals, in order to identify,responsively to the signals, respective regions of the array on whichthe beams of light pulses reflected from the target scene are incident;and processing the signals from the sensing elements in the identifiedregions in order determine respective times of arrival of the lightpulses.
 11. The method according to claim 10, wherein processing thesignals comprises grouping the sensing elements in each of theidentified regions together to define super-pixels, and to processingthe signals from the sensing elements in each of the super-pixelstogether in order to determine the respective times of arrival.
 12. Themethod according to claim 10, wherein processing the signals comprises,after identifying the respective regions, actuating only the sensingelements in each of the identified regions, while the remaining sensingelements in the array are inactive.
 13. The method according to claim10, wherein identifying the respective regions comprises selecting thecandidate regions based on nominal design values together with assemblytolerances and operational tolerances.
 14. The method according to claim10, wherein the radiation source comprises at least one vertical-cavitysurface-emitting laser (VCSEL).
 15. The method according to claim 14,wherein the at least one VCSEL comprises an array of VCSELs.
 16. Themethod according to claim 14, wherein the sensing elements comprisesingle-photon avalanche diodes (SPADs).
 17. A method for depth sensing,comprising: driving a radiation source to emit a first number of beamsof light pulses toward a target scene; imaging the target scene onto anarray of a second number of sensing elements, configured to outputsignals indicative of respective times of incidence of photons on thesensing element, wherein the first number is greater than one, and thesecond number exceeds the first number; finding an initial set ofcandidate regions of the array on which the beams of light pulsesreflected from the target scene are incident; calculating a model, basedon the initial set, that predicts locations of additional regions of thearray on which the beams of light pulses reflected from the target sceneare expected to be incident; searching over the locations predicted bythe model while operating the radiation source and receiving the signalsin order to identify an additional set of regions of the array on whichthe light pulses reflected from the target scene are incident; andprocessing the signals from the sensing elements in the initial andadditional sets of regions in order determine respective times ofarrival of the light pulses.
 18. The method according to claim 17,calculating the model comprises adjusting the model iteratively until anumber of the regions from which the signals are received exceeds apreset threshold.
 19. The method according to claim 18, whereinadjusting the model comprises adding the identified additional regionsto the initial set to produce a new set of the regions, and updating themodel based on the new set.
 20. The method according to claim 17,wherein the model is selected from a group of types of models consistingof a homographic model, a quadratic model, and a low-order spline.